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(54) Method and apparatus for retrieving multimedia data using shape information 



(57) A method and apparatus for retrieving multime- 
dia data using shape information. The multimedia data 
retrieval apparatus comprises a shape information fea- 
ture extractor for receiving shape information of a query 
image and extracting a feature of the received shape 
information using a shape information descriptor based 
on eigen vectors of a multilayer covarlance matrix, an 
image feature extractor for extracting a feature of each 
image data in the same manner as the shape informa- 
tion feature extractor, a multimedia database creator for 
creating a multimedia database on the basis of the fea- 
tures extracted by tiie Image feature extractor, a similar- 
ity comparator for comparing the feature of the query 
image extracted by the shape information feature 
extractor with the features of the image data in the mul- 
timedia database to calculate similarities therebetween, 
and a compared result output unit for outputting the 
results calculated by the similarity comparator. Multime- 
dia data can accurately and rapidly be retrieved by 
defining and using a shape information descriptor with a 
consistency against rotation, scaling and translation of 
the shape information. 
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Description 

BACKGROUND OF THE INVENTION 
5 Field of the Invention 

10001 ] The present invention relates in general to a method and apparatus for retrieving multimedia data by extract- 
ing a feature of shape information of an image using eigen vectors of a shape information covariance matrix and calcu- 
lating a similarity on the basis of the extracted feature, and more particularly to a method and apparatus for retrieving 
10 multimedia data, in which the multimedia data can rapidly and accurately be retrieved by using multilayer eigen vectors 
capable of expressing complex shape information in detail and having a consistency against even rotation, scaling and 
translation of an image. 

Description of the Prior Art 



[0002] Up to now, a language-based search has mostly been used for data retrieval. However, recently, massive 
multimedia data composed of movies, synthetic images, still images, voice, moving images, music and others as well 
as characters has been present on Internet or a multimedia datat)ase according to developments of the Internet and 
multimedia, resulting in a need for retrieving such nnultimedia data. This has keenly required an effective retrieval 
20 method capable of readily retrieving multimedia data desired by the user from bulky data on the Internet or multimedia 
database. 

[0003] Multimedia data is much larger In size than data composed of only characters, and it is a combination of var- 
ious types of information such as images, sounds, characters, etc. As a result, it is next to impossible to retrieve desired 
multimedia data using the multimedia data itself. For this reason, in order to retrieve multimedia data from a multimedia 

25 database, respective multimedia data expressible features are previously extracted through a preprocessing procedure 
and then compared respectively with information in the multimedia database. For example, in the case of retrieving 
video with a mixture of images, voice and audio, respective features of the images, voice and audio are extracted and 
then calculated in similarity with information in a multimedia database to be retrieved. As a result, desired information 
can be retrieved In accordance with the similarity calculation. In this regard, key points in the multimedia data retrieval 

30 are the types of features of multimedia data to be considered, how to express the features and how to compare between 
the features. Herein, a data model expressive of each feature is called a descriptor. 

[0004] A still image or moving image retrieval method is new most studied in multimedia data retrieval techniques. 
In such a retrieval method, features of an image, such as a color, texture, shape, etc. are extracted and then measured 
in similarity. For example, used as descriptors expressive of the color feature may be a color histogram, correlogram, 

35 etc. [see: J. Huang, S. R. Kumar, M. Mrtra. W. J. Zhu, and R. Zabih. Image indexing using color correlation, Proc, 16th 
IEEE Conf. on computer Vision and Pattern Recognition, pp. 762-768, 1997]. Further, a wavelet coefficient, DFT coef- 
ficient, etc. may be used as descriptors expressive of the texture feature. In other words, various descriptors may be 
used to express one feature and have both merits and demerits. In this connection, the performance of a retriever may 
be greatly Influenced by an employed descriptor. 

40 [0005] A shape information retrieval method is one of useful methods for image retrieval. Herein, shape information 
of an object signifies information indicating which pixel of an arbitrary image belongs to the object and which pixel of the 
arbitrary image belongs to a background. For an effective shape information retrieval, it Is necessary to define a 
descriptor capable of appropriately expressing shape information of an object and compare a similarity of the shape 
information on the basis of the defined descriptor. Existing descriptors used for the shape information retrieval may gen- 

45 erally be classified into two types, or geometric feature-based descriptors and moment feature-based descriptors. The 
geometric feature-based descriptors may generally be a parameter, area, maximum radius, minimum radius, corner, 
roundness, etc., and the moment feature-based descriptors be a center of mass, orientation, bounding rectangle, best- 
fit ellipse, eigen vector, etc. For the purpose of accurately and rapidly retrieving image data, the above shape informa- 
tion descriptors should be consistent regardless of any variation of an Image such as rotation, scaling, translation, etc. 

50 of an object. 

[0006] At present, the multimedia data retrieval still stays at the initial stage. One of existing shape information fea- 
ture extraction methods Is to use eigen vectors of a covariance matrix of shape information. As shown in Fig. 1 , the 
eigen vectors of the covariance matrix are composed of two vectors capable of expressing a distribution of the shape 
information. The two eigen vectors have their directions signifying two axes (i.e., major and minor axes) Indicative of dis- 
ss tribution directions of the shape information, respectively, and their magnitudes representing distribution degrees of the 
shape information, respectively. Here, the major axis represents a main distribution direction of the shape information, 
and the minor axis represents a minimum distribution direction of the shape information. 

[0007] Defining a covariance matrix C as in the below equation 1, eigen vectors of the covariance matrix can be 
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calculated in the following manner: 



[Equation 1] 



c = 



10 



c c 



IS [0008] A center of mass (mx.my) of the shape information can be expressed as in the below equation 2: 

^ N ^ N 

'"x = 7v Z ^/ = A/ Z yi [Equation 2] 

20 

[0009] In the above equation 2, "N" indicates the total number of pixels in the shape information, and "Xj" and 
indicate the position of an ith pixel. Calculating the center of mass as In the above equation 2. respective components 
^xx* Cxy srid Cyx of the covarlance matrix can expressed by the following equation 3: 

25 

1 

^xx yv Z (>^r^x)(^r^x) [Equation 3] 

1 " 

^xy = /vZ (^/■'"xKyr'"/) 
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[001 0] In the above equation 3, the conrrponents Cxx and Cyy of the covarlance matrix indicate x-axis and y-axis dis- 
tribution degrees of the shape information, respectively, and the components C^y and Cyx of the covarlance matrix indl- 
45 cate a correlation between x and y coordinates. 

[0011] Defining eigen vectors of the covariance matrix C obtained in the above manner respectively as Ai and A2 
and eigen values of the cpvariance matrix C respectively as r^ and X2* the following equation 4 is established therebe- 
tween: 
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[Equation 4] 
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[00121 Consequently, the eigen vectors and and eigen values and of the covariance matrix C can be 
obtained by solving the above equation 4. As mentioned above, the eigen vectors A^ and Ag of the covariance matrix 
C represent the main and minimum distribution directions of the shape information, respectively, and the eigen values 
-, and r2 of the covariance matrix C represent the distribution degrees of the shape information in the mam and mini- 
mum distribution directions, respectively. 

10013] The above-mentioned shape information feature extraction method using the eigen vectors of the covari- 
ance matrix is able to express an approximate distribution of the shape information with a small amount of data and thus 
has advantages in that it is small in calculation amount, simple in calculation algorithm and has a consistency against 
translation of the shape information. However, the above-mentioned shape information feature extraction method is dis- 
advantageous in that it has a limitation in accurate shape information expression because it should express the entire 
shape information using only two eigen vectors in a single layer. In other words, eigen vectors to be calculated with 
respect to different types of shape information may often have the same value, resulting in a grievous situation. Further, 
the eigen vectors have no consistency against scaling or rotation of the shape information. As a result, the eigen vectors 
are insufficient to define a descriptor for expression of the shape information, leading to a reduction in the accuracy of 
the associated multimedia data retrieval method. 
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SUMMARY OF THE INVENTION 

[0014] Therefore, the present invention has been made in view of the above problems, and it is an object of the 
present invention to provide a method and apparatus for retrieving multimedia data, in which the multimedia data can 
accurately and rapidly be retrieved by adopting a shape information feature extraction method and apparatus capable 
of expressing complex shape information in detail using multilayer eigen vectors of a covariance matrix for the definition 
of a descriptor of the shape information. 

[001 5] It is another object of the present invention to provide a method and apparatus for retrieving multimedia data, 
in which the multimedia data can accurately and rapidly be retrieved by defining and using a shape Information descrip- 
tor with a consistency against rotation, scaling and translation of an object. 

[001 6] In accordance with one aspect of the present invention, the above and other objects can be accomplished 
by a provision of a method for retrieving multimedia data using shape information, comprising the first step of receiving 
shape information of a query image and extracting a feature of the received shape information using a shape informa- 
tion descriptor based on eigen vectors of a multilayer covariance matrix; the second step of extracting a feature of each 
image data in the same manner as the above first step; the third step of creating a multimedia database on the basis of 
the features extracted at the above second step; the fourth step of comparing the feature of the query image with the 
features of the image data in the multimedia database to calculate similarities therebetween; and the fifth step of out- 
putting the results calculated at the above fourth step. 

[001 7] In accordance with another aspect of the present invention, there is provided an apparatus for retrieving mul- 
timedia data using shape Information, which Is capable of embodying the above multimedia data retrieval method. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0018] The above and other objects, features and advantages of the present Invention will be more clearly under- 
stood from the following detailed description taken in conjunction with the accompanying drawings. In which: 
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Fig. 1 is a graph illustrating eigen vectors of a covariance matrix (CMEVs); 

Fig. 2 is a schematic block diagram of a multimedia data retrieval apparatus in accordance with the present inven- 
tion; 

Fig. 3 is a flowchart illustrating a multilayer CMEV-based shape Information feature extraction step in a multimedia 
5 data retrieval method in accordance with the present invention: 

Figs. 4a and 4b are views illustrating a region partition procedure for obtaining multilayer CMEVs in accordance 
. with the present invention; 

Figs. 5a and 5b are views illustrating examples of shape information feature extraction based on an eigen vector 
angle in accordance with the present invention; 
10 Figs. 6a and 6b are views illustrating examples of shape information feature extraction based on a center point in 
accordance with the present invention; 

Figs. 7a to 7f are views illustrating a shape information feature extraction procedure based on a compactness in 
accordance with the present invention; 

Figs. 8a and 8b are views illustrating examples of shape information feature extraction based on the compactness 
IS in accordance with the present invention; 

Fig. 9 is a block diagram of a multilayer CMEV-based shape information feature extractor in the multimedia data 
retrieval apparatus of Fig. 2; 

Fig. 10 is a detailed block diagram of feature extraction means in the shape information feature extractor of Fig. 9; 
and 

20 Fig. 11 is a block diagram of a similarity comparator in the multimedia data retrieval apparatus of Fig. 2. 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0019] With reference to Fig. 2, there is schematically shown in block form the construction of a multimedia data 
25 retrieval apparatus in accordance with the present invention. In the present invention, a multimedia data retrieval 
method based on the multimedia data retrieval apparatus of Fig. 2 first receives shape information of a query image and 
extracts a feature of the received shape information using a shape information descriptor based on eigen vectors of a 
multilayer covariance matrix. Then, the multimedia data retrieval method extracts a feature of each image data in the 
same manner as the above step and creates a multimedia database on the basis of the extracted features. Subse- 
30 quently, the multimedia data retrieval method compares the feature of the query image with the features of the image 
data in the created multimedia database to calculate similarities therebetween and outputs the calculated results. 
Therefore, the multimedia retrieval method of the present invention is characterized by the step of extracting a feature 
of shape Information using a shape information descriptor defined In a proper manner and the step of comparing a sim- 
ilarity between two images on the basis of the extracted feature. 
35 [0020] According to the present invention, the shape information feature extraction step is characterized in that a 
shape information descriptor is defined by applying eigen vectors of a covariance matrix (refenred to hereinafter as 
CMEVs) to every layer. 

[0021] In a different manner from the conventional technique of expressing shape information using only two eigen 
vectors, the present method partitions the shape information into a plurality of regions according to layers and obtains 
40 CMEVs for each of the partitioned regions. Because the obtained eigen vectors express shape information of the asso- 
ciated partitioned regions, respectively, they are increased in number in proportion to the number of the partitioned 
regions so that they can express the shape information in more detail. Therefore, the present invention is able to 
express complex shape information in more detail by applying the CMEVs to every layer. 

[0022] Fig. 3 is a flowchart illustrating the multilayer CMEV-based shape information feature extraction step in the 
45 multimedia data retrieval method in accordance with the present Invention. First, CMEVs are obtained from the entire 
region of shape information at step SI and a feature of the entire region of the shape information is extracted on the 
basis of the obtained CMEVs at step S2. Then, the entire region of the shape information is partitioned into a plurality 
of regions on the basis of two axis of the obtained CMEVs at step S3. It is determined at step S4 whether the partitioned 
regions belong to the last layer If the partitioned regions do not belong to the last layer, the operation returns to the 
50 above step SI to obtain CMEVs from the partitioned regions and extract features of the partitioned regions on the basis 
of the obtained CMEVs. Here, the last layer is deterntined according to the number of layers which is arbitrarily set to 
obtain information minutely to a degree desired by the user. 

[0023] Although the present invention is mainly described with respect to a binary mask as binary shape informa- 
tion, it may be applied to any other object expressible element such as a contour representing the boundary between a 
55 background and an object. 

[0024] Figs. 4a and 4b are views illustrating a region partition procedure for obtaining multilayer CMEVs in accord- 
ance with the present invention. Fig. 4a shows the first region partition. As shown in this drawing. CMEVs are obtained 
from the entire region of input shape information and two axes (major and minor axes) indicative of a distribution of the 
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shape information are obtained from the CMEVs. As a result, the entire region of the shape information is partitioned 
into four regions R1, R2. R3 and R4 due to the intersection of the two axes. Subsequently, each of the partitioned 
regions R1 R2 R3 and R4 is repartttioned as in the above manner. Hence, as shown in Fig. 4b. the region R1 is parti- 
tioned into regions R5. R6. R7 and R8 and the remaining regions R2. R3 and R4 are similarly partitioned into regions 
R9-R20 A smaller number of partitioned layers leads to a smaller number of separated regions. Namely, as partitioned 
layers are increased in number, partitioned regions are similarly Increased in number, thereby giving a more detailed 
expression of the shape information. In the preferred embodiment of the present invention, the number of layers may 
arbitrarily be set to express the shape information minutely to a degree desired by the user. 

[0025] It should be noted that a shape information descriptor must appropriately be defined for the extraction of fea- 
tures of the shape information from the partitioned regions. In the prefenred embodiment of the present invention, the 
shape information descriptor may be defined by a ratio of magnitudes (eigen values) of two eigen vectors of a covan- 
ance matrix, an eigen vector angle, a center point or a compactness, as will hereinafter be mentioned in detail. 

(1) RATIO OF MAGNITUDES OF TWO EIGEN VEr inRR feiaen ration 

[0026] In the preferred embodiment of the present invention, a feature of shape information may be expressed by 
a ratio of magnitudes (eigen values) of eigen vectors of a covariance matrix forming two axes. In this case, the ratio of 
the magnitudes of the two eigen vectors may be a ratio of the magnitude of the eigen vector of the minor axis to the 
magnitude of the eigen vector of the major axis or vice versa. 

10027] The reason why the eigen values of the major and minor axes themselves are not used but the ratio thereof 
is used as a shape information descriptor in the present invention is that the descriptor must have a consistency against 
scaling of the shape information. In other words, provided that eigen values are obtained from shape information with 
the same shape and different sizes, they will have a difference therebetween corresponding to the scaling of the shape 
Information. However, the obtained eigen value ratio can be consistent regardless of the scaling of the shape informa- 
tion because the shape information Is scaled in the same ratio with respect to both the major and minor axes. 

(P ) FIGEN VECTOR ANGLE 

[0028] In expressing a feature of shape information, the use of an eigen vector direction in a partitioned region as 
the feature of the shape information makes it possible to express a contour oriented property in the partitioned region, 
as will hereinafter be described with reference to Figs. 5a and 5b. 

[0029] Fig. 5a shows an example where elliptical shape information is partitioned into two layers in accordance with 
the present invention, and Fig. 5b shows an example where rectangular shape information is partitioned into two layers 
in accorclance with the present invention. Eigen vector magnitude ratios and angles obtained in the first layers are 

35 nearly the same in both cases. Accordingly, both types of shape information can be equally featured when they are 
expressed by only an eigen vector magnitude ratio in one layer. However, four eigen vector magnitude ratios obtained 
in the second layer of Fig. 5a are analogous to those of Fig. 5b. but eigen vector angles are different in both cases. 
Namely it can be seen from Fig. 5a that the directions of eigen vectors in the second layer are slanted toward a contour. 
In this manner, the use of an eigen vector angle other than an eigen vector magnitude ratio as a shape information 

40 descriptor makes it possible to identify the different types of shape information as shown in Figs. 5a and 5b and accu- 
rately extract features of those shape information. 

[0030] Preferably, the eigen vector angle may be an angle of the eigen vector indicative of the major axis or an 
angle of the eigen vector indicative of the minor axis. 

[0031 ] In the case where an eigen vector direction is used as a shape information descriptor, the associated eigen 
45 vector angle may be defined by an absolute value of a difference between an eigen vector angle obtained in the previ- 
ous layer and an eigen vector angle obtained in the current layer. This allows the shape information descriptor to have 
a consistency against rotation of the shape information. 



50 



(S) CENTER POINT (center points 



[0032] In the preferred embodiment of the present invention, a center point of each region other than the above- 
mentioned eigen vector magnitude ratio and angle may be used as a shape information descriptor for the shape infor- 
mation feature extraction. For example, in the case where different types of shape information have the same eigen vec- 
tor magnitude ratio and angle as shown in Figs. 6a and 6b. it Is impossible to express features of the shape information 
55 on the basis of only the above-mentioned eigen vector magnitude ratio and angle. For this reason, it is preferred that a 
center point of each partitioned region is used as a shape information descriptor to make a distinction between the fea- 
tures of the shape information in Figs. 6a and 6b. In Fig. 6a. a center point in each of the partitioned regions is far apart 
from a center of mass in the first layer. But. in Fig. 6b. a center of mass in the second layer is near the center of mass 
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in the first layer. In this regard, the features of the shape information in Figs. 6a and 6b can be differently expressed by 
the center point. 

[0033] According to the present Invention, the center point can be defined by any one of the following four methods. 

5 1 ) The center point can be defined by relative coordinate values (x and y coordinate values) from a center of mass 

obtained in the first layer. 

2) In order to ensure a consistency against rotation of shape Information, the center point can be defined by relative 
coordinate values (x and y coordinate values) when two eigen vectors obtained in the first layer are set as two axes. 

3) In order to ensure a consistency against rotation and scaling of shape information, the center point can be 
10 defined by values obtained by dividing coordinate values (x and y coordinate values) when two eigen vectors 

obtained in the first layer are set as two axes, by an eigen value of the major or minor axis in the first layer or any 
other value with a consistency against the scaling of the shape information. 

4) In order to ensure a simple calculation and a consistency against rotation and scaling of shape information, the 
center point can be defined by a value obtained by dividing a distance from a center of mass ot>tained in the first 

15 layer to a center of mass of each region by an eigen value of a major or minor axis obtained in the first layer or any 
other reference value. This fourth method is more advantageous than the first to third methods in that the center 
point is defined by one value about the distance instead of two values about the x and y coordinates. 



(4) COMPACTNESS 

20 

[0034] In the preferred embodiment of the present invention, a compactness may be obtained with respect to each 
region of a multilayer to express a feature of shape information. Preferably, the compactness may be defined by a ratio 
of the number of pixels In an object region to the number of pixels in the entire region. 

[0035] Figs. 7a to 7f are views illustrating a shape information feature extraction procedure based on the compact- 
25 ness in accordance with the present invention. Figs. 7a and 7b show an example where the compactness is obtained 
in the first layer. In the preferred embodiment of the present invention, the compactness may be defined by a ratio of the 
number of pixels In an object region of each layer to the number of pixels in a reference region of the associated layer 
instead of a ratio of the number of pixels in the original object region to the number of pixels in the entire region. In this 
case, the compactness may have different values according to reference region determination methods. A reference 
30 region in the first layer is defined by a minimum rectangular region which is parallel with two axes of eigen vectors 
obtained in the first layer and fully contains an object region in the first layer. In Fig. 7b, the reference numeral TO 
denotes the reference region in the first layer. In Fig. 7d. the reference numerals Tl. T2, T3 and T4 denote reference 
regions in the second layer, respectively. As shown in Fig. 7d, the reference region TO Is partitioned into the reference 
regions T1 . T2. T3 and T4 by the two eigen vectors. In the same manner, reference regions in the third layer are defined 
35 as indicated by the reference numerals T5-T20 in Fig. 71 Defining the reference regions in the above manner, the com- 
pactness in each of the reference regions can be calculated as in the following equation: 



compact(i) = R |/T j 

40 where. "R** represents the number of pixels in an object region, "T" represents the number of pixels in a reference 

region and "i" represents a number of either the object region or reference region. 

[0036] The reason why the compactness is used as a shape information descriptor in the present invention is that 
it can express, in distinction from each other, features of different types of shape information which have like eigen vec- 
tor magnitude ratios but different shapes as shown in Figs. 8a and 8b. In other words, the compactness in Fig. 8a is 

45 higher than that in Fig. 8b, thereby making a distinction between two images. 

[0037] In the multimedia data retrieval method of the present invention, the step of comparing the feature of the 
query image with the features of the Image data in the multimedia database to calculate similarities therebetween 
includes the step of calculating a difference eigen_ratio_diff between eigen vector magnitude ratios, a difference 
angle_diff between eigen vector angles, a difference center_diff between center points and a difference compact_diff 

50 between compactnesses with respect to each region of the query image and image data. 

[0038] In the case where a shape information descriptor is defined at the shape information feature extraction step 
by an absolute value of a difference between an eigen vector angle in the previous layer and an eigen vector angle in 
the current layer to have a consistency against rotation of shape information, a similarity comparison operation for the 
first layer is performed in a different manner from that for the subsequent layers at the similarity comparison step. That 

55 is, an eigen vector angle and a center point are used for the second layer and over, not for the first layer. 

[0039] The similarity between two types of shape information can be compared on the basis of a match rate 
match_rate therebetween, which can be calculated as in the following equation 5. In the preferred embodiment of the 
present invention, a lower match rate indicates that two images are more similar and a higher match rate indicates that 



BNSDOCID: <EP 1 026601 A2J_> 



10 



IS 



20 



55 



EP 1 026 eOl A2 

the two images are more different. 



W-1 



T! • eigen_ratio_diff(n)+W2 • angle _diff(n)+W2 • center _diff(n)-¥ 
match_rate = ^ [Equation 5] 

• compact_dlff(n)) 

N : number of regions 

(= ^ 4' ( A = number oj layers) 

l-O 

: weights 



eigen_ratio_dfff{n)=\eigen_ratio{f f^)-elgen_ratio{s ^)\ 
anfir/e_d/ff(n)=oan5f/e(/o)-angle(f„)|-|angle(So)-angIe(s„)|i 
25 center _diff{n)=\ center{ f „)- center{s „)| 

compact_diff{n)=\ compact{ f^y compact{s „)| 

f/ : /fih feature of first image 
30 Sf : itii Mature of second image 

[0040] In the above equation 5. "N" represents the total number of regions, or the sum of the region numbers In 
respective layers, and "L" represents the number of layers. Also. "f|" and "Sj" represent features of Ith regions of two Input 
Images to be compared, respectively, and signify weights applied respectively to descriptor differences 

35 {eigen_ratio_diff. angle_diff. centerjdiff and compact_dlfff) at the similarity comparison step. 

[0041 ] At the similarity comparison step, the difference eigen_ratio_diff between the eigen vector magnitude ratios, 
the difference angle_diff between the eigen vector angles, the difference center_diff between the center points and the 
difference compactjdiff between the compactnesses are calculated and summed. Alternatively, the weights may be set 
differently according to features of input shape information and the similarity may be measured on the basis of the 

40 resultant weights. For example, in the case where an eigen vector angle is used for the similarity comparison, the prob- 
ability that the eigen vector angle will have a different value irrespective of shape Information of a binary image becomes 
higher as an eigen vector magnitude ratio is approximated to 1 . As a result, in this case, It is necessary to apply a weight 
to an eigen vector angle difference. 

[0042] In the present invention, the eigen vector magnitude ratio difference eigen_ratiojdiff is defined by an abso- 
45 lute value of a difference between an eigen vector magnitude ratio of first shape information and an eigen vector mag- 
nitude ratio of second shape Information. 

[0043] In one embodiment of the present invention, in the case where a shape information descriptor is defined by 
an eigen vector angle, the eigen vector angle difference angle_diff is defined by an absolute value of a difference 
between an absolute value of a difference between eigen vector angles in the previous and current layers of the first 
50 shape information and an absolute value of a difference between eigen vector angles in the previous and current layers 
of the second shape information. In an alternative embodiment of the present invention, in the case where a shape 
Information descriptor is defined by an absolute value of a difference between an eigen vector angle in the previous 
layer and an eigen vector angle In the current layer, the eigen vector angle difference angle_diff is defined by an abso- 
lute value of a difference between two eigen vector angles as in the below equation 6: 



angle_diff(n) - \ angle(f - angle(s I [Equation 6] 

[0044] In another embodiment of the present invention, the eigen vector angle difference angle_diff can be defined 
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as in the below equation 7: 

anglejcliff(n) = 1 1 angle(f - angle(f „) | - 1 angleCs^) - angleCs^) 1 1 X (1 - eigen_ratio(f „)) [Equation 7] 

5 [0045] In the case where the eigen vector magnitude ratio eigen_ratio is approximated to 1 . the eigen vector angle 
may be changed sensitively to a variation of shape information. In this regard, a smaller weight is applied to the eigen 
vector angle difference angle_diff if the eigen vector magnitude ratio eigen_ratio is approximated to 1 , and a larger 
weight is applied to the eigen vector angle difference angle_diff if the eigen vector magnitude ratio eigen_ratio Is 
approximated to 0. At this time, an eigen vector magnitude ratio eigen_ratio(fn) in the above equation 7 is used as a 

10 weight control factor. Alternatively, as the case may be, an eigen vector magnitude ratio eigen_ratio(Sn) may be used as 
the weight control factor or a combination of the eigen vector magnitude ratio eigen_ratio(fn) and eigen vector magni- 
tude ratio eigen_ratio(Sn) may be used as the weight control factor. 

[0046] The center point difference centerjdiff is defined by an absolute of a difference between two center points, 
and the compactness difference compact_diff is defined by an absolute value of a difference between two compact- 
15 nesses. 

[0047] Now, a detailed description will be given of the multimedia data retrieval apparatus in accordance with the 
present invention. 

[0048] Referring again to Fig. 2, the multimedia data retrieval apparatus comprises a shape information feature 
extractor 1 1 for receiving shape information of a query image and extracting a feature of the received shape information 
20 using a shape information descriptor based on eigen vectors of a multilayer covariance matrix. An image feature extrac- 
tor 22 is adapted to extract a feature of each image data in the same manner as the shape information feature extractor 
11 . A multimedia datattase creator 33 is adapted to create a multimedia database on the basis of the features of the 
image data extracted by the image feature extractor 22. The multimedia data retrieval apparatus further comprises a 
similarity comparator 44 for comparing the feature of the query image extracted by the shape information feature extrac- 
ts tor 1 1 with the features of the image data in the created multimedia database to calculate similarities therebetween, and 
a compared result output unit 55 for outputting the results calculated by the similarity comparator 44. For the multimedia 
data retrieval by the present apparatus, the shape information feature extractor 1 1 receives shape information of a 
query image and extracts a feature of the received shape information using a shape information descriptor based on 
eigen vectors of a multilayer covariance matrix. For the creation of a multimedia database, the image feature extractor 
30 22 extracts a feature of each image data in the same manner as the shape information feature extractor 1 1 . and the 
multimedia database creator 33 creates the multimedia database on the basis of the features of the image data 
extracted by the image feature extractor 22. The similarity conrparator 44 compares the feature of the query image 
extracted by the shape information feature extractor 11 with the features of the image data in the created multimedia 
database to calculate similarities therebetween. Then, the compared result output unit 55 outputs the results calculated 
35 by the similarity comparator 44. 

[0049] In the multimedia data retrieval apparatus of the present invention, the shape information feature extractor 
1 1 and image feature extractor 22 are the same In construction, which will hereinafter be mentioned in detail with refer- 
ence to Fig. 9. 

[0050] Referring to Fig. 9, there is shown in block form the construction of each of the shape information feature 

40 extractor 1 1 and image feature extractor 22 in the multimedia data retrieval apparatus of the present invention. As 
shown in this drawing, each of the shape information feature extractor 1 1 and image feature extractor 22 includes a first 
switch 10 for making a distinction between the initial input image and an output image from region partition means 
60 and transferring the initial input image to covariance matrix calculation means 20 if a given layer is the initial layer 
and the output image from the region partition means 60 to the covariance matrix calculation means 20 if the given layer 

45 is not the initial layer. The covariance matrix calculation means 20 is adapted to calculate a covariance matrix for shape 
information of the image transferred from the first switch Swi 10. Each of the shape information feature extractor 1 1 and 
image feature extractor 22 further Includes eigen vector calculation means 30 for calculating eigen vectors of the cov- 
ariance matrix calculated by the covariance matrix calculation means 20. and feature extraction means 40 for extracting 
a feature of the shape information of the image transferred from the first switch Swi 10 on the basis of the eigen vectors 

50 calculated by the eigen vector calculation means 30. The region partition means 60 is adapted to partition the shape 
information of the image transferred from the first switch S^i 1 0 into four regions according to two axes of the eigen vec- 
tors calculated by the eigen vector calculation means 30 if the given layer is not the last layer. Each of the shape infor- 
mation feature extractor 1 1 and image feature extractor 22 further includes a second switch 8^2 50 for transferring the 
feature extracted by the feature extraction means 40 to an output terminal if the given layer is the last layer and to the 

55 region partition means 60 if the given layer is not the last layer. 

[0051] Fig. 10 is a detailed block diagram of the feature extraction means 40 in the shape information feature 
extractor 1 1 or image feature extractor 22 of Fig. 9. In the preferred embodiment of the present invention, the feature 
extraction means 40 may include any one or a combination of an eigen vector magnitude ratio calculator (referred to 
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hereinafter as eigen_ratio calculator) 41 . an eigen vector angle calculator (referred to hereinafter as angle calculator) 
42, a center point calculator (referred to hereinafter as center_j)oint calculator) 43 and a compactness calculator 
(referred to hereinafter as compact calculator) 44. Most preferably, the feature extraction means 40 may include all of 
the eigen_ratio calculator 41. angle calculator 42. center_point calculator 43 and compact calculator 44. In this case, 

5 the feature extraction means 40 extracts features such as an eigen vector magnitude ratio, an eigen vector angle, a 
center point and a compactness from the shape information of the image transferred from the first switch S^i 1 0 on the 
basis of the eigen vectors calculated by the eigen vector calculation means 30. The feature extraction means 40 further 
Includes a feature combiner 400 for combining the extracted features into a desired format and outputting the combined 
result to the second switch 8^2- 

10 [0052] Preferably, the above calculators In the feature extraction means 40 may be provided as means for realizing 
the respective embodiments at the shape information feature extraction step in the multimedia data retrieval method of 
the present invention as stated previously. 

[0053] Next, a detailed description wilt be given of the operation of the shape information feature extractor 11 in the 
multimedia data retrieval apparatus of the present Invention. 

IS [0054] First, upon receiving an Image of the first layer, the first switch S^i 10 transfers the received image to the 
covariance matrix calculation means 20, which then calculates a covariance matrix for shape information of the trans- 
ferred image. The eigen vector calculation means 30 calculates eigen vectors of the covariance matrix calculated by the 
covariance matrix calculation means 20, and the feature extraction means 40 extracts a feature of the shape Informa- 
tion of the image transferred from the first switch S^i 1 0 on the basis of the eigen vectors calculated by the eigen vector 

20 calculation means 30. The second switch 8^2 50 determines whether a given layer is the last layer. If the given layer Is 
not the last layer, the second switch 50 transfers the feature extracted by the feature extraction means 40 to the 
region partition means 60. Then, the region partition means 60 partitions the shape information of the image transferred 
from the switch Swi 10 Into four regions according to two axes of the eigen vectors calculated by the eigen vector cal- 
culation means 30 and transfers an image of each of the partitioned regions to the covariance matrix calculation means 

2S 20 through the first switch S^-i 10. Subsequently, the covariance matrix calculation means 20 calculates a covariance 
matrix for shape information of the transferred image, the eigen vector calculation means 30 calculates eigen vectors 
of the covariance matrix calculated by the covariance matrix calculation means 20. and the feature extraction means 40 
extracts a feature of the shape information of the transferred image on the basis of the eigen vectors calculated by the 
eigen vector calculation means 30. This operation Is repeated until the given layer becomes the last layer. When the 

30 given layer Is the last layer, the second switch 8^2 50 transfers the feature extracted by the feature extraction means 40 
to the output terminal. 

[0055] Rg. 1 1 is a block diagram of the similarity comparator 44 in the multimedia data retrieval apparatus of Rg. 
2. As shown in this drawing, the similarity comparator 44 includes a first feature separator 1 0 1 for separating the feature 
of the query image extracted by the shape Information feature extractor 1 1 Into an eigen vector magnitude ratio, eigen 

35 vector angle, center point and compactness, and a second feature separator 102 for separating each of the features of 
the Image data in the created multimedia database into an eigen vector magnitude ratio, eigen vector angle, center 
point and compactness. The similarity comparator 44 further includes a calculator (referred to hereinafter as 
elgen_ratio_dlff calculator) 201 for calculating a difference between the eigen vector magnitude ratios from the first and 
second feature separators 101 and 102. a calculator (referred to hereinafter as anglejdiff calculator) 202 for calculating 

40 a difference between the eigen vector angles from the first and second feature separators 101 and 102, a calculator 
(referred to hereinafter as center_dfff calculator) 203 for calculating a difference between the center points from the first 
and second feature separators 101 and 102, and a calculator (referred to hereinafter as compact_diff calculator) 204 
for calculating a difference between the compactnesses from the first and second feature separators 101 and 102. The 
similarity comparator 44 further Includes first to fourth weight calculators 300-303 for calculating weights to the eigen 

45 vector magnitude ratio difference from the eigen_ratio_diff calculator 201, the eigen vector angle difference from the 
angle_diff calculator 202, the center point difference from the center_diff calculator 203 and the compactness difference 
from the compact_diff calculator 204. respectively The similarity comparator 44 further includes an adder 500 for add- 
ing the weighted eigen vector magnitude ratio difference, eigen vector angle difference, center point difference and 
compactness difference from the first to fourth weight calculators 300-303, and a similarity calculator 600 for calculating 

50 the similarity between the feature of the query image extracted by the shape information feature extractor 1 1 and each 
of the features of the image data in the created multimedia database on the basis of the result added by the adder 500. 
In the preferred emtxxJiment of the present invention, the similarity comparator 44 may include any one or a combina- 
tion of the eigen_ratio_dlff calculator 201 . anglejdiff calculator 202, centerjdiff calculator 203 and compactjdiff calcu- 
lator 204 according to the presence of the eigen_ratio calculator 41 , angle calculator 42, center_point calculator 43 and 

55 compact calculator 44 in each of the shape information feature extractor 1 1 and image feature extractor 22. Most pref- 
erably the similarity comparator 44 may include all of tine eigen_ratio_diff calculator 201, angle_diff calculator 202, 
center_diff calculator 203 and compactjdiff calculator 204 to express the similarity very precisely. 
[0056] Next, a detailed description will be given of the operation of the similarity comparator 44 with the above-men- 
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tioned construction in accordance with the present invention. 

[0057] First, upon receiving the feature of the query image extracted by the shape information feature extractor 1 1 , 
the first feature separator 101 separates the received feature into an eigen vector magnitude ratio, eigen vector angle, 
center point and compactness. Also, the second feature separator 102 receives each of the features of the image data 

5 in the created multimedia database and separates the received feature into an eigen vector magnitude ratio, eigen vec- 
tor angle, center point and compactness. Then, the eigen_ratio_diff calculator 201 calculates a difference between the 
eigen vector magnitude ratios from the first and second feature separators 101 and 102, and the angle_diff calculator 
202 calculates a difference between the eigen vector angles from the first and second feature separators 101 and 102. 
Also, the center_dlff calculator 203 calculates a difference between the center points from the first and second feature 

10 separators 101 and 102, and the compact_diff calculator 204 calculates a difference between the compactnesses from 
the first and second feature separators 101 and 1 02. The first to fourth weight calculators 300-303 apply proper weights 
to the eigen vector magnitude ratio difference from the eigen_ratio_diff calculator 201 , the eigen vector angle difference 
from the angle_diff calculator 202, the center point difference from the centerjdiff calculator 203 and the compactness 
difference from the compact_diff calculator 204, respectively. Subsequently, the adder 500 adds the weighted eigen 

15 vector magnitude ratio difference, eigen vector angle difference, center point difference and compactness difference 
from the first to fourth weight calculators 300-303. and the similarity calculator 600 calculates the similarity between the 
feature of the query image extracted by the shape information feature extractor 1 1 and each of the features of the image 
data in the created multimedia database on the basis of the result added by the adder 500. Then, the similarity calcu- 
lator 600 provides the calculated result to the compared result output unit 55. 

20 [0058] At present, multidegree moments such as Zernike moments are used as shape Information descriptors [see: 
CHO^HUAKTEN, ROLAND T CHIN, On Image Analysis by the Methods of Moments. IEEE Trans. RAMI, vol. 10, NO. 
4, JULY 1998]. An n-degree moment can be calculated in the form of sum of powers as in the below equation 8: 



[0059] Accordingly, as the degree is higher, the powers are Increased in number, resulting In a complexity In calcu- 
lation. To the contrary, in the present invention, a moment of the first degree is calculated as in the equation 2 to obtain 
a center point as a shape information descriptor. Further, the calculation of a covariance matrix Is composed of only 
multiplication, addition and subtraction as in the equation 4. Therefore, the present invention ensures a smaller amount 

35 of calculation as compared with conventional methods employing moments. Further, the present invention can regulate 
an expression degree of shape information by adjusting the number of layers and ensure a consistency against rotation, 
scaling and translation of the shape information. Furthermore, in the case where the present invention is implemented 
in a hardware manner, simple calculation means such as the covariance matrix calculation means, eigen vector calcu- 
lation means, feature extraction means and region partition means are provided and repeated in operation, thereby 

40 making the hardware construction simple. Therefore, the present invention has the effect of accurately expressing 
shape information with a snnall amount of calculation and a small amount of data to accurately and rapidly retrieve 
desired multimedia data. 

[0060] Although the preferred embodiments of the present invention have been disclosed for illustrative purposes, 
those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without 
45 departing from the scope and spirit of the invention as disclosed in the accompanying claims. 



25 




[Equation 8] 



where, p •«- q = n. 



30 



Claims 



1 . A method for retrieving multimedia data using shape information, comprising the steps of: 



50 



55 



a) receiving shape information of a query image and detracting a feature of the received shape information 
using a shape information descriptor based on eigen vectors of a multilayer covariance matrix; 

b) extracting a feature of each image data in the same manner as said step a); 

c) creating a multimedia database on the basis of said features extracted at said step b); 

d) comparing said feature of said query image with said features of said image data in said multimedia data- 
base to calculate similarities therebetween; and 

e) outputting the results calculated at said step d). 
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2. The method as set forth in Claim 1 , wherein said step a) includes the steps of: 

a-1) obtaining a covariance matrix of said shape information of said query image; 
a-2) obtaining eigen vectors of said covariance matrix obtained at said step a-1); 
s a-3) extracting said feature of said shape Information of said query Image using a shape Information descriptor 

based on said ergen vectors obtained at said step a-2); 

a-4) partitioning said shape information of said query image Into a plurality of regions on the basis of two axes 
of said ergen vectors obtained at said step a-2); and 

a-5) repeating said steps a-1) to a-4) with respect to each of said regions partitioned at said step a-4) until a 
10 given layer becomes a last layer. 

3. The method as set forth in Claim 1 or Claim 2, wherein said shape Information is contour information of a binary 
image. 

15 4. The method as set forth in Claim 1 or Claim 2. wherein said shape Information descriptor is defined by a ratio of 
magnitudes of said eigen vectors. 

5. The method as set forth in Claim 1 or Claim 2. wherein said shape information descriptor is defined by an angle of 
any one of said eigen vectors. 

20 

6. The method as set forth In Claim 1 or Claim 2, wherein said shape information descriptor is defined by a center 
point in a given region. 

7. The method as set forth in Claim 1 or Claim 2, wherein said shape Information descriptor is defined by a compact- 
's ness. said compactness being a ratio of the number of pixels in an object region to the number of pixels In a refer- 
ence region. 

8. The method as set forth in Claim 1 or Claim 2. wherein said shape information descriptor is defined by a ratio of 
magnitudes of said eigen vectors, an angle of any one of said eigen vectors, a center point in a given region and a 

30 compactness In the given region. 

9. The method as set forth in Claim 4, wherein said two axes of said eigen vectors are major and minor axes and said 
eigen vector magnitude ratio .is a ratio of the magnitude of said eigen vector of said minor axis to the magnitude of 
said eigen vector of said major axis. 

35 

10. The method as set forth in Claim 4, wherein said two axes of said eigen vectors are major and minor axes and said 
eigen vector magnitude ratio is a ratio of the magnitude of said eigen vector of said major axis to the magnitude of 
said eigen vector of said minor axis. 

40 11, The method as set forth in Claim 5, wherein said two axes of said eigen vectors are major and minor axes and said 
eigen vector angle is an angle of said eigen vector of said major axis. 

12. The method as set forth in Claim 5. wherein said two axes of said eigen vectors are major and minor axes and said 
eigen vector angle is an angle of said eigen vector of said minor axis. 

45 

13. The method as set forth In Claim 5, wherein said eigen vector angle is defined by an absolute value of a difference 
between an eigen vector angle obtained In a previous layer and an eigen vector angle obtained in a cun-ent layer. 

14. The method as set forth In Claim 6, wherein said center point is defined by relative coordinate values from a center 
50 of mass in a first layer. 

15. The method as set forth in Claim 6. wherein said center point is defined by relative coordinate values when a center 
of mass In a first layer is set as an origin and two eigen vectors in said first layer are set respectively as major and 
minor axes. 



55 



16. The method as set forth In Claim 6, wherein said center point is defined by values obtained by dividing coordinate 
values when a center of mass in a first layer is set as an origin and two eigen vectors in said first layer are set 
respectively as major and minor axes, by an eigen value of said major or minor axis in said first layer or any other 
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reference value. 

17. The method as set forth in Claim 7, wherein said reference region is defined in a first layer by a minimum rectan- 
gular region which is parallel with two eigen vectors obtained in said first layer and fully contains said object region 

5 in said first layer, said reference region being partitioned into a plurality of regions by said eigen vectors to define 

reference regions in a second layer and over. 

18. The method as set forth in Claim 1 or Claim 2, wherein said step d) includes the step of calculating said similarity 
between said feature of said query image and each of said features of said image data in said multimedia database 

10 on the basis of any one of a difference between eigen vector magnitude ratios of said query image and image data, 
a difference between eigen vector angles of said query image and Image data, a difference between center points 
of said query Image and image data and a difference between compactnesses of said query image and image 
data. 

IS 19. The method as set forth in Claim 18, wherein said eigen vector angle difference is defined by an absolute value of 
a difference between an eigen vector angle In a previous layer and an eigen vector angle in a current layer. 

20. The method as set forth In Claim 1 8. wherein said eigen vector magnitude ratio difference Is defined by an absolute 
value of the difference between said eigen vector magnitude ratios of said query Image and image data. 

20 

21 . The method as set forth in Claim 1 8, wherein said step includes the step of applying a weight to said eigen vector 
angle difference on the basis of said eigen vector magnitude ratios if said eigen vector magnitude ratios are approx- 
imated to 1. 

25 22. The method as set forth in Claim 18. wherein said center point difference is defined by an absolute value of the dif- 
ference between said center points of said query image and Image data. 

23. The method as set forth In Claim 18, wherein said compactness difference is defined by an absolute value of the 
difference between said compactnesses of said query image and image data. 

30 

24. An apparatus for retrieving multimedia data using shape Information, comprising: 

shape information feature extraction means for receiving shape information of a query image and extracting a 
feature of the received shape information using a shape information descriptor based on eigen vectors of a 
35 multilayer covariance matrix; 

Image feature extraction means for extracting a feature of each Image data in the same manner as said shape 
information feature extraction means; 

multimedia database creation means for creating a multimedia database on the basis of said features extracted 
by said image feature extraction means; 
40 similarity comparison means for comparing said feature of said query image extracted by said shape informa- 

tion feature extraction means with said features of said image data in said multimedia database to calculate 
similarities therebetween; arKi 

compared result output means for outputting the results calculated by said similarity comparison means. 

45 25. The apparatus as set forth in Claim 24. wherein said shape information feature extraction means includes: 

a first switch for making a distinction between an Initial Irput Image and an output image from region partition 
means and transferring said initial input image to covariance matrix calculation means if a given layer is an Ini- 
tial layer and said output image from said region partition means to said covariance matrix calculation means 
50 if said given layer is not said initial layer; 

said covariance matrix calculation means calculating a covariance matrix for shape information of the image 
transferred from said first switch ; 

eigen vector calculation means for calculating eigen vectors of said covariance matrix calculated by said cov- 
ariance matrix calculation means; 
55 feature extraction means for extracting a feature of said shape information of said image transferred from said 

first switch on the basis of said eigen vectors calculated by said eigen vector calculation means; 
said region partition means partitioning said shape information of said image transferred from said first switch 
into a plurality of regions according to two axes of said eigen vectors calculated by said eigen vector calculation 
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means if said given layer is not a last layer; and 

a second switch for transferring said feature extracted by said feature extraction means to an output terminal if 
said given layer is said last layer and to said region partition means if said given layer is not said last layer. 

26. The apparatus as set forth In Claim 24 or Claim 25, wherein said shape information is contour information of a 
binary image. 

27. The apparatus as set forth in Claim 24 or Claim 25, wherein said feature extraction means includes a calculator for 
calculating a ratio of magnitudes of said eigen vectors. 

28. The apparatus as set forth in Claim 24 or Claim 25. wherein said feature extraction means includes a calculator for 
calculating an angle of any one of said eigen vectors. 

29. The apparatus as set, forth in Claim 24 or Claim 25, wherein said feature extraction means includes a calculator for 
IS calculating a center point in a given region. 

30. The apparatus as set forth in Claim 24 or Claim 25, wherein said feature extraction means includes a calculator for 
calculating a compactness, said compactness being a ratio of the number of pixels in an object region to the 
number of pixels in a reference region. 



10 



20 



31 . The apparatus as set forth in Claim 24 or Claim 25. wherein said feature extraction means includes: 



a first calculator for calculating a ratio of magnitudes of said eigen vectors; 
a second calculator for calculating an angle of any one of said eigen vectors; 
25 a third calculator for calculating a center point in a given region; and 

a fourth calculator for calculating a compactness in the given region. 

32. The apparatus as set forth in Claim 27. wherein said two axes of said eigen vectors are major and minor axes and 
said eigen vector magnitude ratio calculator is adapted to calculate a ratio of the magnitude of said eigen vector of 

30 said minor axis to the magnitude of said eigen vector of said major axis. 

33. The apparatus as set forth in Claim 27, wherein said two axes of said eigen vectors are major and minor axes and 
said eigen vector magnitude ratio calculator is adapted to calculate a ratio of the magnitude of said eigen vector of 
said major axis to the magnitude of said eigen vector of said minor axis. 

35 

34. The apparatus as set forth in Claim 28, wherein said two axes of said eigen vectors are major and minor axes and 
said eigen vector angle calculator is adapted to calculate an angle of said eigen vector of said major axis. 

35. The apparatus as set forth in Claim 28, wherein said two axes of said eigen vectors are major and minor axes and 
40 said eigen vector angle calculator is adapted to calculate an angle of said eigen vector of said minor axis. 

36. The apparatus as set forth in Claim 28. wherein said eigen vector angle calculator is adapted to calculate an abso- 
lute value of a difference between an eigen vector angle obtained in a previous layer and an eigen vector angle 
obtained in a current layer. 

45 

37. The apparatus as set forth in Claim 29. wherein said center point calculator is adapted to obtain relative coordinate 
values from a center of mass in a previous layer. 

38. The apparatus as set forth in Claim 29. wherein said center point calculator is adapted to obtain relative coordinate 
so values when a center of mass in a first layer is set as an origin and two eigen vectors in said first layer are set 

respectively as major and minor axes. 

39. The apparatus as set forth in Claim 29, wherein said center point calculator is adapted to divide coordinate values 
when a center of mass in a first layer is set as an origin and two eigen vectors in said first layer are set respectively 

55 as major and minor axes, by an eigen value of said major or minor axis in said first layer. 

40. The apparatus as set forth in Claim 29, wherein said center point calculator is adapted to divide a distance from a 
center of mass obtained in a first layer to a center of mass of each region by an eigen value of a major or minor axis 
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obtained in said first layer or any other reference value. 

41 . The apparatus as set forth in Claim 30. wherein said reference region Is defined in a first layer by a minimum rec- 
tangular region which is parallel with two eigen vectors obtained In said first layer and fully contains said object 

5 region in said first layer, said reference region being partitioned into a plurality of regions by said eigen vectors to 

define reference regions in a second layer and over. 

42. The apparatus as set forth in Claim 24. wherein said similarity comparison means includes: 

10 a first feature separator for separating said feature of said query image extracted by said shape information fea- 

ture extraction means into an eigen vector magnitude ratio, an eigen vector angle, a center point and a com- 
pactness; 

a second feature separator for separating each of said features of said image data In said multimedia database 
into an eigen vector magnitude ratio, an eigen vector angle, a center point and a compactness: 
IS a first difference calculator for calculating a difference between said eigen vector magnitude ratios from said 

first and second feature separators; 

a second difference calculator for calculating a difference between said eigen vector angles from said first and 
second feature separators; 

a third difference calculator for calculating a difference between said center points from said first and second 
20 feature separators; 

a fourth difference calculator for calculating a difference between said compactnesses from said first and sec- 
ond feature separators; 

first to fourth weight calculators for calculating weights to said eigen vector magnitude ratio difference, eigen 
vector angle difference, center point difference and compactness difference from said first to fourth difference 

25 calculators, respectively; 

an adder for adding the weighted eigen vector magnitude ratio difference, eigen vector angle difference, center 
point difference and compactness difference from said first to fourth weight calculators; and 
a similarity calculator for calculating said similarity between said feature of said query image extracted by said 
shape information feature extraction means and each of said features of said image data in said multimedia 

30 database on the basis of the result added by said adder. 

43. The apparatus as set forth in Claim 42. wherein said secorxJ difference calculator is adapted to calculate an ak>so- 
lute value of a difference between an absolute value of a difference between eigen vector angles in previous and 
current layers of said query image and an absolute value of a difference between eigen vector angles in previous 

35 and current layers of said image data. 

44. The apparatus as set forth in Claim 42. wherein said first difference calculator is adapted to calculate an absolute 
value of the difference between said eigen vector magnitude ratios of said query image and image data. 

40 45. The apparatus as set forth in Claim 42, wherein said third difference calculator is adapted to calculate an absolute 
value of the difference between said center points of said query image and image data. 

46. The apparatus as set forth in Claim 42, wherein said fourth difference calculator is adapted to calculate an ak>solute 
value of the difference between said compactnesses of said query image and image data. 

45 
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(a) elliptical binary shape information (b) rectangular binary shape information 
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(a) object region in first layer 




(c) object regions in second layer 




(e) object regions in third layer 




(b) reference region in first layer 
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(d) reference regions In second layer 
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Fig . 8a 




Fig . 8b 
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